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The problems of conducting research in schools are discussed. Solutions to 
social psychological problems are suggested, and an improved design for research is 
recommended. Clearer definition or terminology is recommended to avoid the 
confusion that can result from ambiguous terms. A research design should be 
adaptable to the administrative and instructional styles of individual schools. The 
researcher should observe the proper hierarchy in gaining permission to conduct 
research and should report results immediately. Deadline dates and specifications of 
event sequences with names of responsible persons are requisites for successful 
interpersonal relations. The following steps are recommended to avoid the frequent 
failure of educational research to improve educational practice: formulation, 
instructional specifications, component preparation and tryout, product preparation, 
quality verification, and operations analysis. CBS) 
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Conducting Research in School Systems^ 

Richard E. Schutz 
Southv T est Regional Laboratory 
for Educational Research and Development 

Dr. Hayman addressed you this morning from the vantage point of a 
person who has seen many researchers thrown out of schools. I follow him 
as a researcher who has been thrown out of many schools. Although our 
experiential bases differ , I presently see things much the same way John 
does. As John implied, the bounce of a researcher is usually figural 
rather than literal, but the consequences are similar-- inaccurate or in- 
complete data. 

I must admit that the problems Dr. Hayman spoke to were not the sort 
I anticipated when I agreed to discuss designing research to meet the prob- 

i 

lems he raised. The social psychological problems that John enumerated are 
not conventionally included in discussions of either reading research or 
general research methodology. In fact one is hard-pressed to find any 
admission of their existence. Yet, I concur completely that interpersonal 
relations are necessary, if not sufficient, considerations in conducting 
research in school systems. One cannot meet such problems through research 
design; people design is required. Ideally, I’d recommend one try to avoid 
rather than meet interpersonal problems. But since John was brave enough 
to raise social psychological problems I will try to deal with such aspects 
of them as I can before returning to more familiar design territory. You 
will note that while I accept Dr. Hayman's identification of the determinants 
of the problems, I disagree with some of his suggested solutions. 



j I 



i 




\ 

.1 





1 



Paper presented at the Annual Convention of the International Reading 



Association, Boston, April 26, 1968 



U. S. DEPARTMENT OF HEALTH, EDUCATION & WELFARE 
OFFICE OF EDUCATION 



THIS DOCUMENT HAS BEEN REPRODUCED EXACTLY AS RECEIVED FROM T 
PERSON OR ORGANIZATION ORIGINATING IT. POINTS OF VIEW OR OPINIOR 

* mesm 0fFICIAL om « of" ssss 




US 






I 

i 

* 

- 2 - 

As Dr. Hayman implied, a school researcher must develop educational 

I 

I 

theological competence as well as scientific competence. Education cur- 

l 

reiitly rests professionally on a theological rather than a theoretical 

i ' i 

foundation. It is easy for a researcher to stumble unwittingly over some 
currently hallowed dogma. For example, I've learned to use Piaget and 
Bruner as positive word weapons, like mother and apple pie, as well as data 
references. "Inquiry learning" and "interpersonal sensitivity" are also 
currently on the side of good rather than evil in most quarters. But you 
have to be careful to respond to denominational slogan peculiarities within 
the theological framework. Phonics in some school districts is very good, 
in other districts neutral or negative. 

i 

In general, the school researcher must live with a conceptual depriva- 

t 

tion which his predecessors have cheerfully condoned and nourished. This 
conceptual deprivation is far more debilitating than cultural deprivation. 

: Terms like "comprehension," "dyslexia," and "blending," to mention only a 

few, do have meaning. But the associated referents are either other ab- 

: 

stractions or ambigious operations. The apparent specificity such terms 
| provide is misleading at best. Consider the term "bilingual," which is a 

focus of a good deal of educational, social, and political attention at 
the moment. The term incorporates both dialect and literacy considerations. 
Permuting these, Joseph Follettie in our Laboratory has distinguished 48 
different instructional outcome referenced options ranging from Monolingual, 
standard dialect, illiterate as one condition to bilingual, bidialectic, 
standard and substandard Spanish, standard and substandard English, bi- 
| literate. With one term referring to 48 different conditions, one inevi- 

I 

| tably encounters communication difficulties among professionals. The 
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solution lies In cleaning up conceptual relationships rather than in 
cleaning up interpersonal relationships. Thus the most direct route is via 

logical analysis rather than psychoanalysis. 

As John so clearly impressed upon us, the school is not a research 
institution. The responsibilities of the teacher differ from those of the 
researcher. This makes reading research particularly difficult. Since 
reading is almost universally regarded as a high priority objective, school 
people naturally feel a high degree of responsibility for "protecting the 
child" in this area. Thus, even when all concerned recognize that overall 
instructional effectiveness in reading is low, there is a general reluctance 
to introduce any manipulations for fear things will deteriorate even further. 
This "drowning man" syndrome is particularly evident in reading research as 

compared to research in other school subjects. 

John recommends "successful compromise" to resolve such problems. I 
would prefer negotiation to compromise. Design validity is akin to pregnancy. 
It is the researcher's responsibility to generate a completely clean 
design that is feasible to the responsibile school people. The operational 
responsibilities of the schools preclude their bending much. If the 
researcher cannot produce a research design that permits him to generate 
the observations he requires in a manner the school people can manage, the 
quicker he folds his tent the better for all concerned. It is neither 
reasonable nor feasible to attempt to conduct all reading research under 

natural school conditions. 

I have found that districts vary tremendously in their administrative 
and instructional styles. In some districts it is possible for a researcher 
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to "get by" with much less attention to interpersonal relations than in 
others. However, I have found the following "before .and after" professional 
courtesies are always in good taste. First, a school district is a highly 
developed bureaucracy. Start at the top with the superintendent's office. 

You may never have to see the superintendent per se. But the superintendent 
has to have an opportunity to know what's going on. If you don't provide 
him this opportunity and anything goes wrong, expect no mercy from anyone 
and expect that whoever goes into the district to request research coopera- 
tion the next time will encounter increased resistance. Similar respect 
for whatever administrative red-tape the district imposes is a necessity. 

A researcher seldom enjoys this process, but it is a necessary price for 
continuous cooperation. Consider "after." A researcher is inclined to have 
a short memory after a study is completed. The preparation of the research 
report may take some time, and it is seldom prepared in a form that is 
comprehend ible to school people. It is indeed "extra work" to report back 
either in oral or written form the findings of the study as soon ls possible 
after the study is completed. But unless this is done, the school people 
have a perfect right to feel that they were simply "used" to advance the 
personal interests of the researcher. Professional courtesy demands better 
treatment of school people than most researchers like to extend. I've found 
that school people typically respond beyond the call of professional duty 
when one treats them as intelligent human beings rather than as "sources for 
subjects." A researcher would resei w . being viewed merely as a source for 
"design and analysis." 

Note that treating school people as people is only a prerequisite. 
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Dale Carnegie is no substitute for Sir Ronald Fisher. Dr. Hayman comes 
perilously close at times to implying that interpersonal relations are ends 
rather than enablers. For example: "Teachers need to identify with the 

[ project and its purposes. They must be involved to the extent possible in 

designing experimental procedures and in planning for the implementation of 
procedures which are necessarily designed by someone else. They must be 

made to feel an integral part of what is happening." 

Teachers must not only feel a part of what is happening, they are what 
is happening. This does not mean, however, that they have a contribution 
to make at every stage of the research. Education needs further, rather 
' than less, specialized division of labor. Many authorities in teacher ed- 

ucation and graduate training equate cooperation with contribution. It is 

» 

professionally sad that the skills of many teachers and researchers are 
| indistinguishable when the roles of each have become distinctly more complicated 

since the days in which many school people received their formal professional 
training. To insist that a teacher and/or principal be present or involved 

: in every aspect of a study is an unreasonable concession to educational 

; * • 

bureaucracy. 

| | The educational research community, of course, has a matching bureau- 

cracy. I can attest from personal experience and from observing others that 

! 

throwing together a team of say a teacher, a statistician, a psychometrician, 

; I 

;! a liberal arts scholar, and a curriculum specialist and expecting something 

! to get done, is a wishful expectation. These roles are essential ingredients, 

j but converting roles into outcomes requires a management and technological 

; sophistication that we have not yet achieved in education. 
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Dr. Haymaa devoted considerable attention to "problems of communication." 
I must admit that I feel that the term, "communications" is getting tired 
from over-use. Or more accurately, I'm getting tired of hearing "communica- 
tion problem" used as a pseudo-explanation of so many current problems. The 
explanation is too inclusive and abstract. With language an inevitable 
concomitant of human endeavors, one can smugly cite a "communications 
problem" for anything that goes wrong in any sector of human interaction. 

On the other hand, I heartily endorse ambiguity reduction in communication. 

As steps in this direction I recommend deadline dates, and the specification 
of ordered sequences of events accompanied By specific names of persons 

responsible for specific requirements. I also highly recommend trying out 

# 

any written communication on a couple of people to test the variability of 
the reactions it evokes. I've found that other people are remarkably 
reliable in their misunderstanding of my crystal clear communications. It's 
my validity problem, not their reliability. 

Dr. Hayman is equivocal on the concept of feedback: "Seldom will the 

curriculum materials and the instructional techniques being researched be 
perfect and changes may be needed while the project is in progress. 

Making changes during the course of a project violates the classical experi- 
ment notion, of course, but may be a necessary trade-off to assure that 
children receive the most effective instruction possible." The distinction 
which Michael Scriven makes between formative and summative evaluation 
should help John's guilt in this area. There is an impressive research 
tradition in such productive fields as chemistry and biology for "making 
changes in the course of a project" as a means of evaluating a phenomenon 
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during some stage of development. Moreover, there are established procedures 

for I insuring the dependability of such systematic or initially adventitious 

* 

probes. For a treatment of such procedures I recommend Murray Sidman's 
Tactics of Scientific Research . The end result expected from such manipula- 
tions is an improved product — in our context more dependably produced 
instructional outcomes. This contrasts with summative evaluation which 

relates to the evaluation of finally developed educational programs. Here 

* 

the expected end result is a set of descriptive statements about a single 
program or about the relative merits of two or more programs. Even in a 
summative evaluation context it is still perfectly feasible "to make changes 
needed while the project is in progress." The only requirement is that 
whatever procedures and materials were used be accurately and fully des- 
cribed. One has a completely free hand in science at least the way I 
play it -- so long as he maintains a record which permits replication of 

whatever he does. 

One final quibbly solution before I present a big-problem solution, 
which casts all of the foregoing in a worms-eye perspective. This relates 
to what John labels the particularly vexing problem in obtaining a suitable 
sample. The inconsistency is reflected in the following sentence "The 
researcher will ordinarily be concerned with the behavior of individuals, 
and he will therefore want to select his sample by individuals." Wrong. 

I guess the best reference to why is the Campbell and Stanley chapter in 
the Handbook of Research on Teaching . Simply because a researcher is inter- 
ested in individuals is no rationale for treating instruction as if it were 
individualized. So long as instruction is under the direction of a teacher 
with the class working as a group, rather than completely independently at 
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all times, the unit of interest is by definition the class, not the individual 
pupil. This, however, is in most ways a gain rather than a loss. For j 

example, it is much more feasible to randomly assign intact classes to 
experimental treatements than it is to assign individuals at random. The 
effect of so doing on the statistical significance levels which John worries 
about is not so great as he implies, since one picks up added reliability 
in statistically analyzing mean scores rather than individual scores. 

Thus, we find that good school people and good statisticians happily come 
out quite close in their recommendations on this point. j 

The use of the class rather than the individual as the experimental 

unit also reduces the differential mortality problem which John raises. j 

i I 

It's not a complete solution, and again I refer you to the Campbell-Stanley 

t 

discussion. I would point out, however, that analysis of covariance, which ! 

John states can help out, cannot help out. 

So much for my solutions to the problems John raised. John's concerns j 

w ith interpersonal relations and common courtesy led him to avoid raising | 

i 

• j 

one nasty, but inescapable problem--the utility of educational research. . 

We have all nurtured the polite fiction for too long that educational 

research is a direct route to educational improvement. There is no 

evidence to support the contention that educational research leads to 

improved educational practice. Nor does such a relationship hold for fields , 

other than education. Research is transformed into a practical, useful form 

via operations conventionally labeled development. While there are elaborate 

and sophisticated technologies of development in other fields, in education 

we use the term R&D as if it were a single concept, with the activities 
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j encompassed by the term almost exclusively research rather than development. 

Please note that 1 am in no way disparaging the value of educational research. 
It is simply that research is directed toward increased knowledge rather 
than improved practice. The increased knowledge resulting from research 
may be useful in producing products with social utility, but it is not 
equivalent to such products. The distinction between research, development, 

1 and school practice is analogous to the distinction between conducting an 

aerodynamics experiment in the physics laboratory, developing the SST, and 
piloting a 707. Here it is obvious that different operations are involved. 

But education has created no such division of function and responsibility. 
Physics, engineering, and piloting are not interchangeable, but educational 
| research, development, and instruction are often considered as if they were. 

I 

To give you some indication of the nature of development as I'm using 
the term, I should like to describe the instructional development procedures 
being followed at the Southwest Regional Laboratory for Educational Research 
' and Development. I won't be describing the Laboratory program, only the 

| procedures we make use of in applying what we refer to as the self-correcting 

mechanism to instructional product development. That is, we take it for 
! granted that no materials and procedures, either in education or in other 

fields, will work perfectly the first time they are tried out. An iterative 
j sequence of trial-revision cycles is req, «' 1 to successively eliminate de- 

i fects until the product functions at an appropriate effectiveness level. 1 

I ' will be describing the procedures briefly, but it seems to me that they may 

j represent the basis for a generalized solution to the problems of conducting 

| research in the schools* 

! 
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Formulation represents the first stage in "closing the gap" between 
research and practice. There are always many new and attractive "ideas 
available, but they always require some work to generate a base for further 
instructional development. Formulation usually begins with conceptual 
activities to identify the most powerful manipulable variables in an area 
which can then be woven into a broad instructional strategy. These con- 
ceptual activities should culminate in a set of instructional specifications. 

Instructional specifications include, but are not limited to, sequenced 
statements of desired instructional outcomes. Each objective must also be 
accompanied by specifications for requisite entry behavior, by specifications 
for testing if the outcome is attained, by specifications for the media by 
which the instruction is to be presented, the specific conditions under which 
practice must be given to assure the behavior does occur, and inappropriate 
conditions under which the behavior must not occur. This is a much more 
complicated analytic task than simply stating a terminal behavioral objective 
which is difficult enough in education. A terminal objective, however, repre 
sents the beginning rather than the end of instructional development. 

The component preparation and tryout involves the initial tryout of 
component materials and methods which relate to various aspects of the 
instructional specifications. Often the prototype materials and methods 
are in a "mockup" form which only approximates the expected finished 
product. However, any prototype must be in a form amenable to replication; 
the designer may not specify that a "creative" or "well-educated" teacher 
is required without indicating precisely how such creative behavior can be 
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involves three kinds of activity. First, the 



The tryout of components 
administration of criterion-referenced tests to a representative target 
sample to obtain some indication of the difficulty of attaining the 
instructional outcomes with the target learner population. Second, it is 
usually desirable at this stage to try out instructional components with a 
single learner or a single group of learners. Any prototype instruction 
is inevitably based on implicit initial assumptions about both the target 
population and instructional components. Both university professors and 
classroom teachers tend to invest instruction with greater meaning than it 
possesses for a child since they typically assume an isomorphic relation- 
ship between their perception of a given segment of instruction and the 
child’s perception of the same situation. For example, the simple simulta 
neous presentation of a blue triangle and a red circle may be seen as 
teaching classification, color, geometry, set theory, visual perception, 
arithmetic readiness, inquiry behavior, inference, and a host of other 
high prestige terms. The greater the quantity of abstractions the adult 
can attach to the instructional stimuli, the happier and more educated he 
assumes the child will be. The initial tryout under simulated conditions 
tests these assumptions and provides a practical and efficient basis for 
working out a number of "bugs" and for identifying the child s reactions 
to the instruction. The earlier this initial tryout occurs, the better. 

Third, the component tryout provides an occasion for experimentation 
with important identifiable separable aspects of the instruction, such as 
the nature of illustrations, book size, clarity of directions, and 
presentation mode. 
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Product preparation involves combining and extending the prototype 
components into instruction suitable for classroom tryout* This involves 
more than assemblyline operations. It also involves the introduction of 
motivational devices -- fun and games -- to make the instruction attractive 
to pupils and manageable for teachers. This is not easy to do. It is 
possible to produce highly concentrated drill and practice which has no 
mo 'vational appeal. It is also possible to produce highly motivating fun 
and games with no instructional value. The technology for producing 
instruction which is both highly efficient and highly motivating is still 

primitive. 

Quality Verification involves the successive trial-revision cycles 
required to bring a product to an acceptable level of performance under 
complex, real-world conditions. As I've already indicated, no product is 
likely to function adequately the first time it is tried out. Several 
corrective cycles are required to identify and eliminate defects until the 
product adequately satisfies the user's needs. Quality verification tryouts 
are performed by regular school personnel under standard school conditions. 
Such tryouts are not demonstrations; special visibility is minimized. 

A final step which we call operations analysis refers to the successive 
refinement of the foregoing steps in instructional development. This is 
almost absent in American education. Yet it is the principal yield of 
science and technology in other sectors of life where the invention of the 
method of invention occurred during the nineteenth century. With an 
effective operations analysis one always has his "next generation" product 
underway before his "current generation" product is fully installed. This 
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progressive 



ive acc um ulation of efficiency requires both a criterion~referenced 



outlook and a continuous development operation which has not heretofore 
existed in education. It is in the operations analysis area where it seems 
to me that "curriculum" divisions of school systems could function most 
effectively if they possessed appropriate skills and orientation. 

In sum, there are indeed unique problems associated with conducting 



research more efficiently outside the classroom. Others can be smoothed 
out by greater attention to interpersonal relations and more creative 
research designs. Bigger and better solutions are being created by in- 
creased investments of time and energy in the new empirical frontier of 
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research in schools. Many of these can be avoided by conducting the 



educational development. 
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